Statistical Curvature and Stochastic Complexity

We discuss the relationship between the statistical embedding curvature [1, 2] and the logarithmic regret [11] (regret for short) of the Bayesian prediction strategy (or coding strategy) for curved exponential families and Markov models. The regret of a strategy is defined as the difference of the logarithmic loss (code length) incurred by the strategy and that of the best strategy for each data sequence among a considered class of prediction strategies. (The considered class is referred to as a reference class.) Since a prediction strategy is equivalent to a probability distribution, the class of prediction strategy is equivalent to a statistical model. Note that the logarithmic loss (equivalent to code length) by the minimax strategy is equal to Rissanen’s stochastic complexity (SC). SC is generalization of Minimum Description Length [8, 3] and plays an important role in statistical inference such as model selection, universal prediction, universal coding, etc. For this matter, it can be shown that the Bayesian strategy with Jeffreys prior (Jeffreys strategy for short) asymptotically achieves SC upto the constant term, when the reference class is an exponential family[12, 13, 16]. This is due to the fact that the logarithmic loss of Bayes mixture strategy is affected by the exponential curvature of the considered class. Hence, the Jeffreys strategy does not achieve the SC in general, if the reference class is not an exponential family. For a curved exponential family case, in order to obtain the minimax regret, we give a method to modify the Jeffreys mixture by assuming a prior distribution on the exponential family in which the curved family is embedded. We also consider the expected version of regret (known as redundancy in information theory field). When the true probability distribution belongs to the reference class, the Jeffreys strategy asymptotically achieves the minimax redundancy, irrelevant to the curvature of the reference class as shown by Clarke and Barron [6]. However, if the true probability distribution does not belong to the reference class, the situation differs and the redundancy of Jeffreys strategy is affected by both exponential and mixture curvatures of the reference class. Finally, we study the exponential curvature of a class of Markov sources defined by a context tree (tree model). Tree models are classified to FSMX models and non FSMX models. It is known that FSMX models are exponential families in asymptotic sense. We are interested in the problem if non FSMX models are exponential families or not. We show that a certain kind of non FSMX tree model is curved in terms of exponential curvature.